In order to solve the problem that the Single Gaussian Model (SGM) for background could not adapt to non-stationary scenes and the "ghost" phenomenon due to sudden moving of a motionless object. An SGM for background using block-based gradient and linear prediction was put forward. Firstly, SGM was implemented on the pixel level and updated adaptively according to the changes of the pixels' values, at the same time the frame was processed by the block-based gradient algorithm, obtaining the background by judging whether the gradient of sub-block was within the threshold value and eliminating "ghost"; and then foreground from the block-based gradient algorithm and that from the SGM were made "AND" operation, improving the judgment of the background in non-stationary scenes; lastly the linear prediction was employed to process the foreground acquired from the previous operation, resetting the connected regions whose area was less than the threshold value as the background. Simulation experiments were conducted on the CDNET 2012 dataset and Wallflower dataset. In the scenes which varied by a large margin, the accuracy of the proposed method was 40% higher than that of the Gaussian Mixture Model (GMM) in spite of the fact that the detection rate of the proposed method was lower than that of GMM; but in other scenes, the rate of detection was 10% higher and the accuracy was 25% higher. The simulation results show that the proposed method is able to accommodate to the non-stationary scenes and achieve the goal of wiping the "ghost" off, as well as obtain a better result of the background and more detailed foreground than GMM.